Test Runner

Assuming we have defined our Test Methods (page X) on one or more Testcase Classes (page X), how do we actually cause the test to be run?

How It Works

Each member of the xUnit family of Test Automation Frameworks (page X) provides some form of command-line or graphical application that can be used to run our automated tests and report on the results. The Test Runner uses Test Enumeration (page X), Test Discovery (page X) or Test Selection (page X) to obtain a Composite[GOF] test object. The latter may either be a single Testcase Object (page X), a Test Suite Object (page X) or a Composite test suite (a Suite of Suites (see Test Suite Object).) Because all these objects implement the same interface, the Test Runner need not care whether it is dealing with a single test or a multilevel suite.

Why We Do This

We wouldn't want each test automater to have to provide a special means of running their own test suites. This would only get in the way of running all the tests run by everyone on the team. By providing a standard Test Runner, we encourage developers to make it easy to be able to run tests written by different people. We can also provide different ways of running the same tests.

Implementation Notes

There are several different styles of Test Runner. The most common variations are running tests from within an IDE and running tests from the command line. they all depend on the fact that all Testcase Objects implement a standard interface.

Standard Test Interface

In statically-typed languages like (Java and C#) we typically have an interface type (fully abstract class) that defines the interface that all Testcase Objects and Test Suite Objects must implement. Some languages, like C# and Java 5.0, "mix" the implementation in using class attributes or annotations on the Testcase Class. In dynamically-typed languages this interface may not exist explicitly but each implementation class must implement the standard methods. Typically, the standard test interface include the factory method to construct the Test Suite Object and methods on it to count the available tests and to run the tests.

Variation: Graphical Test Runner

A Graphical Test Runner is typically a desktop application or part of an IDE (either built-in or a plug-in) for running tests; at least one, IeUnit runs inside a web browser rather than an IDE. The most common feature of the Graphical Test Runner is some sort of real-time progress indicator. This typically includes a running count of test failures and errors and often includes a colored progress bar which starts off green and turns red as soon as an error or failure is encountered. Some members of the xUnit family include a graphical Test Tree Explorer as a means to drill-down and run a single test from within a Suite of Suites. Here is the Graphical Test Runner from the JUnit plug-in for Eclipse:

The red bar near the top indicates at least one test has failed. The upper text pane shows a list of test failures and test errors. The lower pane show the traceback from the failed test selected in the upper pane.

Variation: Command-Line Test Runner

Command-Line Test Runners are designed to be used from an OS command line or from batch files or shell scripts. They are very handy when working remotely via remote shells or when running the tests from a build script such as "make", "ant" or from a continuous integration tool such as "Cruise Control".

The following is an example of running an runit (RubyUnit) test from the command line:

>ruby testrunner.rb c:/examples/tests/SmellHandlerTest.rb
Loaded suite SmellHandlerTest
Started
.....Finished in 0.016 seconds.5 tests, 6 assertions, 0 failures, 0 errors
>Exit code: 0
Example CommandLineTestRunnerConsole embedded from Ruby/console.txt

The first line is the invokation at the command prompt. In this example we are only running the tests defined in a single Testcase Class, SmellHandlerTest. The next two lines are the initial feedback as it starts up. The series of dots indicate progress, one per test completed. This particular Command-Line Test Runner replaces the dot with an "E" or an "F" if the test errors or fails. The last 3 lines are the summary statistics that provide an overview of what happened. Typically, the exit code is set to the total number of failing/erroring tests so that a non-zero exit code can be interpretted easily as a build failure when run from an automated build tool.

Variation: File System Test Runner

Some Command-Line Test Runners provide the option of searching a specified directory for all files that are tests and run them all. This automated Testcase Class Discovery (see Test Discovery) avoids the need to build up the Suite of Suites in code (Test Enumeration) and helps avoid Lost Tests (see Production Bugs on page X).

There are also external tools that will search the file system for files matching specific patterns and then invoke an arbitrary command against the matched files. These can be used for invoking the Test Runner from a build tool.

Variation: Test Tree Explorer

Members of the xUnit family that turn each Test Method into a Testcase Object can manipulate the tests easily. Many of them provide a graphical representation of the Suite of Suites and allow the user to select an entire Test Suite Object or a single Testcase Object to run. This eliminates the need to create a Single Test Suite (see Named Test Suite on page X) class to run a single test. Here is the Test Tree Explorer of JUnit shown "popped out" over other Eclipse "views":

The left pane of the IDE is the JUnit view within Eclipse. The progress bar is at the top of the view, the upper pane is the Test Tree Explorer and the lower pane is the traceback for the currently selected test failure. Note that some Test Suite Objects in the Test Tree Explorer are "open" revealing their contents while others are closed down. The colored annotation next to each Testcase Object shows its status; the annotations for each Test Suite Object indicates whether any contained Testcase Objects failed or errored. The Test Suite Object called "Test for com.clrstream.ex8.test" is a Suite of Suites for the package "com.clrstream.ex8.test"; "Test for allJUnitTests" is the top-most Suite of Suites for running all the tests.

Page generated at Wed Feb 09 16:39:47 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved